Robotic Touch Perception

ABSTRACT

An apparatus such as a robot capable of performing goal oriented tasks may include one or more touch sensors to receive touch perception feedback on the location of objects and structures within an environment. A fusion engine may be configured to combine touch perception data with other types of sensor data such as data received from an image or distance sensor. The apparatus may combine distance sensor data with touch sensor data using inference models such as Bayesian inference. The touch sensor may be mounted onto an adjustable arm of a robot. The apparatus may use the data it has received from both a touch sensor and distance sensor to build a map of its environment and perform goal oriented tasks such as cleaning or moving objects.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of the filing dateof U.S. Provisional Application No. 62/089,416, filed on Dec. 9, 2014,entitled “INTELLIGENT MACHINE-BASED CLEAN-UP AND DECLUTTERING OF ANUNSTRUCTURED ENVIRONMENT”, which is hereby incorporated by reference inits entirety. This application also claims priority to and the benefitof the filing date of U.S. Provisional Application No. 62/103,573, filedon Jan. 15, 2015, entitled “CLOUD-BASED ROBOTIC SOFTWARE WITHDOWNLOADABLE APPS”, which is hereby incorporated by reference in itsentirety. Finally, this application claims priority to and the benefitof the filing date of U.S. Provisional Application No. 62/139,172, filedon Mar. 27, 2015, entitled “EXTENSIONS AND APPLICATIONS FOR ROBOTSCAPABLE OF MANIPULATION” all of the above referenced applications arehereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

This patent specification relates to the field of machines andapparatuses such as robots capable of receiving touch sensor data toperform goal oriented tasks using data from both touch and imagesensors.

BACKGROUND

While other science fiction mainstays like space travel, globalinstantaneous communication and self-driving cars have come to fruition,general-purpose household robots have remained out of reach, with eventhe most advanced research robots struggling to do simple tasks likefolding towels.

One set of tasks that would be particularly desirable to automate aretasks related to cleaning. This involves moving objects, and thencleaning surfaces, possibly replacing those objects that belong in theparticular space. This patent specification describes methods andapparatus to perform these functions.

Household robots exist today. Most household robots sold today arerobotic vacuum cleaners. The best of these robots sense theirenvironment for mapping purposes, but all household vacuuming robotslack the ability to manipulate objects on purpose (although they mayaccidentally push some objects around the floor).

Some robotic toys have robotic arms that can manipulate small objects.Generally these robots are limited to remote control operation, orsimple pre-programmed sequences that sometimes follow decision-treesbased on sensor input. At best these toys have very limited perception,such as the ability to follow a line drawn on a piece of paper, or toseek or avoid a light. In no way do these toys perform the desired taskof cleaning a surface or space.

Much more advanced perception and object-recognition has beendemonstrated using research robots including advanced tasks such asfolding towels. The authors of this patent specification are unaware,however, of any robot or other device that demonstrated the task ofautonomously cleaning a space beyond the floor. The nearest prior artknown to the authors include a video of a PR1 research robot appearingto clean a room, which in actual fact is remote-controlled by one of theresearchers. The other prior art known to the authors is a robot knownas “Fetch” built by SRI of Menlo Park, Calif. that can grasp and pick upobjects and identify them. This robot does not include the capability tothen put away the object and does not have any capability to cleansurfaces. One of the techniques described in this patent specificationis touch perception. Some researchers have demonstrated the sense oftouch, often in the context of remote-control applications that includehaptics. However, other than bumper switches on robotic vacuums, theauthors of this patent specification are not aware of any demonstrationof touch perception that is used for autonomous object and/orenvironment recognition.

Therefore, a need exists for novel methods of robotic perception whichmay be used for autonomous object and/or environment recognition. Thereis a further need for novel methods for combining touch sensor data withvisual sensor data to autonomously perform an activity.

BRIEF SUMMARY OF THE INVENTION

It is one object of the present invention to provide an apparatus suchas a robot capable of performing goal oriented tasks. In someembodiments, the apparatus uses one or more touch sensors to receivetouch perception feedback on the location of objects and structureswithin an environment. In preferred embodiments, the apparatus mayinclude a fusion engine to combine touch perception data with othertypes of sensor data such as data received from an image or distancesensor. In some embodiments, the apparatus may combine distance sensordata with touch sensor data using Bayesian inference models. In furtherembodiments, the touch sensor may be mounted onto the arm of a robot todetect level of pressure and direction of pressure. The apparatus maythen use the data it has received from both a touch sensor and adistance sensor to build a map of its environment and perform goaloriented tasks such as cleaning or moving objects.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present invention are illustrated as an exampleand are not limited by the figures of the accompanying drawings, inwhich like references may indicate similar elements and in which:

FIG. 1 depicts a perspective view of an example of a mobile robotapparatus comprising a movable arm according to various embodimentsdescribed herein.

FIG. 2 illustrates a block diagram showing some of the some of theelements an example of a mobile robot apparatus according to variousembodiments described herein.

FIG. 3 shows a block diagram showing some of the elements an example ofa processing unit of a mobile robot apparatus may comprise according tovarious embodiments described herein.

FIG. 4 depicts a perspective view of an example of a movable armaccording to various embodiments described herein.

FIG. 5 illustrates a perspective view of an example of a touch sensorcomprising electromechanical skin according to various embodimentsdescribed herein.

FIG. 6 shows a perspective view of an example of an effector suite whichmay be coupled to a movable arm according to various embodimentsdescribed herein.

FIG. 7 depicts a perspective view of the effector suite of FIG. 6 withthe exemplary accessory mount removed according to various embodimentsdescribed herein.

FIG. 8 illustrates a perspective view of the effector suite of FIG. 6with the exemplary accessory mount and the exemplary touch sensor coverremoved according to various embodiments described herein.

FIG. 9 shows two example scans of an object using discrete twodimensional scanners whose scans may be used for mapping of anenvironment according to various embodiments described herein.

FIG. 10 depicts a block diagram of an example of an object library ofdata objects according to various embodiments described herein.

FIG. 11 illustrates a block diagram illustrating some modules which mayfunction as software rules engines and which may provide robotic touchperception according to various embodiments described herein.

FIG. 12 shows a block diagram of an example of a method for combiningsensor data to perform a goal directed action according to variousembodiments described herein.

DETAILED DESCRIPTION OF THE INVENTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items. As used herein, the singularforms “a,” “an,” and “the” are intended to include the plural forms aswell as the singular forms, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, steps, operations, elements, components, and/or groupsthereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by onehaving ordinary skill in the art to which this invention belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure and will not be interpreted in an idealized or overlyformal sense unless expressly so defined herein. In describing theinvention, it will be understood that a number of techniques and stepsare disclosed. Each of these has individual benefit and each can also beused in conjunction with one or more, or in some cases all, of the otherdisclosed techniques. Accordingly, for the sake of clarity, thisdescription will refrain from repeating every possible combination ofthe individual steps in an unnecessary fashion. Nevertheless, thespecification and claims should be read with the understanding that suchcombinations are entirely within the scope of the invention and theclaims.

New methods of building an environment map of an environment usingrobotic perception discussed herein. In the following description, forpurposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be evident, however, to one skilled in the art that the presentinvention may be practiced without these specific details.

The present disclosure is to be considered as an exemplification of theinvention, and is not intended to limit the invention to the specificembodiments illustrated by the figures or description below.

The present invention will now be described by example and throughreferencing the appended figures representing preferred and alternativeembodiments. FIG. 1 illustrates an example of a mobile robot apparatus(“the robot”) 100 according to various embodiments. In some embodiments,a robot may be any machine or apparatus capable of executing a task. Inthe non-limiting example shown by FIG. 1, the robot 100 comprises a body11 with a movable arm (“the arm”) 200 operably coupled to the body 11.One or more transportation conveyances 12 which may facilitate themovement of the robot 100 across surfaces, such as the ground andfloors, and/or through the air, such as propellers and other airtransportation enabling devices, may also be coupled to the body 11. Abody motor 33 may be coupled to one or more transportation conveyances12 and configured to operate the transportation conveyances 12. Inalternative embodiments, the body 11 may not be movable and the robot100 may be generally stationary with a movable arm 200. A power source32 may provide power to a processing unit 21 which may be in electricalcommunication with the functional elements of the body 11 and/or arm200. The arm 200 may be coupled to the body 11 at a proximal end 211 andcoupled to an effector suite 220 at a distal end 212 with one or moresections 206 and joints 201 connecting the proximal end 211 to thedistal end 212. An accessory 300 may be coupled to the effector suite220, such as to an accessory mount 221, which may be directed intocontact with objects 401 and surfaces which may be found in anenvironment. The robot 100 may also comprise one or more touch sensors35, 36, which may receive touch input which may be used to form a map ofthe environment and the objects 401 and surfaces which may be found inthe environment. Additionally, the robot 100 may comprise a distancesensor 37 which may provide distance information which may also be usedto form a map of the environment and the objects 401 and surfaces whichmay be found in the environment.

FIG. 2 illustrates a block diagram showing some of the some of theelements an example of a mobile robot apparatus 100 according to variousembodiments described herein. In preferred embodiments, a robot 100 maycomprise one or more control inputs 31, processing units 21, powersources 32, body motors 33, arm motors 34, body touch sensors 35, armtouch sensors 36, distance sensors 37, and/or microphones 38. It shouldbe appreciated by those of ordinary skill in the art that FIG. 2 depictsthe robot 100 in an oversimplified manner, and a practical embodimentmay include additional components or elements and suitably configured tosupport known or conventional physical components and operating featuresthat are not described in detail herein.

Optionally, a robot 100 may comprise one or more control inputs 31 thata user may interact with such as turnable control knobs, depressablebutton type switches, slide type switches, rocker type switches, or anyother suitable input that may be used to electrically communicate userinput to the robot 100 such as to the processing unit. For example, acontrol input 31 may comprise a switch that may be manipulated by a userto control a function of the robot 100.

In some embodiments, a robot 100 may comprise a power source 32 whichmay provide electrical power to any component that may requireelectrical power. A power source 32 may comprise a battery, such as alithium ion battery, nickel cadmium battery, alkaline battery, or anyother suitable type of battery, a fuel cell, a capacitor, or any othertype of energy storing and/or electricity releasing device. In furtherembodiments, a power source 32 may comprise a power cord, kinetic orpiezo electric battery charging device, a solar cell or photovoltaiccell, and/or inductive charging or wireless power receiver.

In some embodiments, a robot 100 may comprise one or more body motors 33which may be used to move the robot 100 through one or moretransportation conveyances 12. A body motor 33 may comprise a brushed DCmotor, brushless DC motor, switched reluctance motor, universal motor,AC polyphase squirrel-cage or wound-rotor induction motor, AC SCIMsplit-phase capacitor-start motor, AC SCIM split-phase capacitor-runmotor, AC SCIM split-phase auxiliary start winding motor, AC inductionshaded-pole motor, wound-rotor synchronous motor, hysteresis motor,synchronous reluctance motor, pancake or axial rotor motor, steppermotor, or any other type of electrical or non-electrical motor. One ormore transportation conveyances 12 may be configured to facilitate themovement of the robot 100 across a surface. In some embodiments, atransportation conveyance 11 may comprise a wheel, as shown by theexample of FIG. 1, a caster, a tread or track, a low friction pad orbumper, a low friction plate, a ski, a pontoon, or any other suitabledevice configured to reduce the friction between the robot 100 and thesurface over which it is desired to be moved. In further embodiments, atransportation conveyance 12 may comprise a propeller, miniaturized jetengine, or any other air transportation enabling device which may allowthe robot 100 to fly or function similar to a drone air craft. Infurther embodiments a transportation conveyance 12 may comprise a fin, awater jet, a screw, or any other water transportation enabling devicewhich may allow the robot 100 to move on or below the surface of water.In further embodiments a transportation conveyance 12 may comprise arocket, and ion drive, a gyroscope, or any other space transportationenabling device which may allow the robot 100 to move in space.

In some embodiments, a robot 100 may comprise one or more arm motors 34which may be used to move one or more sections 206 and/or othercomponents of an arm 200. An arm motor 34 may comprise any suitable typeof motor such as may be used for a body motor 33. In preferredembodiments, an arm motor 34 may comprise an actuator which may beoperated by a source of energy, typically electric current, hydraulicfluid pressure, or pneumatic pressure, and converts that energy intomotion. Examples of actuators may include comb drives, digitalmicromirror devices, electric motors, electroactive polymers, hydrauliccylinders, piezoelectric actuators, pneumatic actuators,servomechanisms, thermal bimorphs, screw jacks, or any other type ofhydraulic, pneumatic, electric, mechanical, thermal, and magnetic typeof actuator.

In some embodiments, a robot 100 may comprise one or more body touchsensors 35 and/or arm touch sensors 36. Generally, a touch sensor 35,36, may detect contact between an object or surface and the componentthat the touch sensor 35, 36, is coupled to. In preferred embodiments, atouch sensor 35, 36, may be configured to detect the force or pressurebetween an object or surface and the component that the touch sensor 35,36, is coupled to when the surface or object is in contact with thecomponent that the touch sensor 35, 36, is coupled to. A body touchsensor 35 may be coupled to the body 11 of a robot 100 and may beconfigured to detect contact and/or the pressure during contact betweenan object or surface and the component of the body 11 that the bodytouch sensor 35 is coupled to. An arm touch sensor 36 may be coupled tothe arm 200 of a robot 100 and may be configured to detect contactand/or the pressure during contact between an object or surface and thecomponent of the arm 200 that the arm touch sensor 36 is coupled to.

In some embodiments, a robot 100 may comprise one or more distancesensors 37. A distance sensor 37 may include sensors such as fixed(single beam) or rotating (sweeping) Time-of-Flight (TOF) or structuredlight based laser rangefinders, 3D High Definition LiDAR, 3D FlashLIDAR, 2D or 3D sonar sensors and one or more 2D cameras. Further, adistance sensor 37 may also include Passive thermal infrared sensors,Photocell or reflective sensors, Radar sensors, Reflection of ionisingradiation sensors, Sonar sensors, such as active or passive, Ultrasonicsensors, Fiber optics sensors, Capacitive sensors, Hall effect sensors,or any other sensor able to detect the presence of nearby objects andsurfaces without any physical contact. Generally, a distance sensor 37may comprise any type of sensor which is able to provide informationwhich describes the distance between the distance sensor 37 and thedetected object or surface.

In some embodiments, a robot 100 may comprise one or more microphones38. A microphone 38 may be configured to pick up or record audioinformation from the environment around the robot 100 and preferablyfrom a user speaking to or issuing voice commands to the robot 100. Inpreferred embodiments, a microphone 38 may comprise anyacoustic-to-electric transducer or sensor that converts sound in airinto an electrical signal. In further embodiments, a microphone 38 maycomprise any type of microphone such as electromagnetic inductionmicrophones (dynamic microphones), capacitance change microphones(condenser microphones), and piezoelectricity microphones (piezoelectricmicrophones) to produce an electrical signal from air pressurevariations. In further embodiments, a microphone 38 may be in networkcommunication with a robot 100 and may include microphones that usedigital communication to send information, including microphones inremote controls, smart phones, tablet computers, laptop computers, andother like electronic devices that have the ability to translate soundinto data packets for transmission to the robot 100, such as through awired or wireless connection, including implementations in whichpre-processing is performed by the electronic device comprising themicrophone, including voice recognition, before being communicated tothe robot 100.

FIG. 3 depicts a block diagram showing some of the elements an exampleof a processing unit of a mobile robot apparatus 100 (FIGS. 1 and 2) maycomprise according to various embodiments described herein. In someembodiments and in the present example, the robot 100 can be a digitaldevice that, in terms of hardware architecture, comprises one or moreprocessing units 21 which generally include one or more processors 22,and optionally input/output (I/O) interfaces 30, optional radios 23,data stores 24, and memory 25. It should be appreciated by those ofordinary skill in the art that FIG. 3 depicts the processing unit 21 inan oversimplified manner, and a practical embodiment may includeadditional components or elements and suitably configured processinglogic to support known or conventional operating features that are notdescribed in detail herein. The components and elements (22, 30, 23, 24,and 25) are communicatively coupled via a local interface 26. The localinterface 26 can be, for example but not limited to, one or more busesor other wired or wireless connections, as is known in the art. Thelocal interface 26 can have additional elements, which are omitted forsimplicity, such as controllers, buffers (caches), drivers, repeaters,and receivers, among many others, to enable communications. Further, thelocal interface 26 may include address, control, and/or data connectionsto enable appropriate communications among the aforementionedcomponents.

The processor 22 is a hardware device for executing softwareinstructions. The processor 22 can be any custom made or commerciallyavailable processor, a central processing unit (CPU), an auxiliaryprocessor among several processors associated with the processing unit21, a semiconductor-based microprocessor (in the form of a microchip orchip set), or generally any device for executing software instructions.When the processing unit 21 is in operation, the processor 22 isconfigured to execute software stored within the memory 25, tocommunicate data to and from the memory 25, and to generally controloperations of the robot 100 pursuant to the software instructions. In anexemplary embodiment, the processor 22 may include a mobile optimizedprocessor such as optimized for power consumption and mobileapplications.

The I/O interfaces 30 may include one or more control inputs 31,processing units 21, power sources 32, body motors 33, arm motors 34,body touch sensors 35, arm touch sensors 36, and/or distance sensors 37.I/O interfaces 30 may be used to receive and record environmentalinformation and to control one or more functions of a robot 100 to allowit to interact and move through an environment. The I/O interfaces 30can also include, for example, a serial port, a parallel port, a smallcomputer system interface (SCSI), an infrared (IR) interface, a radiofrequency (RF) interface, a universal serial bus (USB) interface, andthe like which may be used to send and receive data to other electronicdevices such as computer devices, programming units, and controllers.

An optional radio 23 enables wireless communication to an externalaccess device or network. In some embodiments, a radio 23 may operate ona WIFI band and may communicate with one or more electronic devices overa wireless network allowing data to be sent and received by the robot100. Any number of suitable wireless data communication protocols,techniques, or methodologies can be supported by the radio 23,including, without limitation: RF; IrDA (infrared); Bluetooth; ZigBee(and other variants of the IEEE 802.15 protocol); IEEE 802.11 (anyvariation); IEEE 802.16 (WiMAX or any other variation); Direct SequenceSpread Spectrum; Near-Field Communication (NFC); Frequency HoppingSpread Spectrum; Long Term Evolution (LTE); cellular/wireless/cordlesstelecommunication protocols (e.g. 3G/4G, etc.); wireless home networkcommunication protocols; paging network protocols; magnetic induction;satellite data communication protocols; wireless hospital or health carefacility network protocols such as those operating in the WMTS bands;GPRS; proprietary wireless data communication protocols such as variantsof Wireless USB; and any other protocols for wireless communication.

A data store 24 may be used to store data. The data store 24 may includeany of volatile memory elements (e.g., random access memory (RAM, suchas DRAM, SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g.,Flash memory, ROM, hard drive, tape, CDROM, and the like), andcombinations thereof. Moreover, the data store 24 may incorporateelectronic, magnetic, optical, and/or other types of storage media.

The memory 25 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, etc.), and combinations thereof.Moreover, the memory 25 may incorporate electronic, magnetic, optical,and/or other types of storage media. Note that the memory 25 may have adistributed architecture, where various components are situated remotelyfrom one another, but can be accessed by a processor 22. The software inmemory 25 can include one or more software programs, each of whichincludes an ordered listing of executable instructions for implementinglogical functions.

In the example of FIG. 3, the software in the memory system 25 includesa suitable operating system (O/S) 27 and programs 28. The operatingsystem 27 essentially controls the execution of input/output interface30 functions, and provides scheduling, input-output control, file anddata management, memory management, and communication control andrelated services. The operating system 27 may be, for example, LINUX (oranother UNIX variant), Android (available from Google), Symbian OS,RobotOS (ROS), Microsoft Windows CE, Microsoft Windows 7 Mobile, iOS(available from Apple, Inc.), webOS (available from Hewlett Packard),Blackberry OS (Available from Research in Motion), and the like. Theprograms 28 may include various applications, add-ons, etc. configuredto provide end user functionality with the apparatus 100. For example,exemplary programs 28 may include, but not limited to, environmentalvariable analytics, environment mapping, and modulation of input/outputinterface 30 functions. In a typical example as shown in FIG. 11,programs 28 for robotic touch perception may comprise a mapping module600 which may further comprise a mapping engine 601, touch engine 602, adistance engine 603, fusion engine 604, and/or functional equivalents.

Further, many embodiments are described in terms of sequences of actionsto be performed by, for example, elements of a computing device. It willbe recognized that various actions described herein can be performed byspecific circuits (e.g., application specific integrated circuits(ASICs)), by program instructions being executed by one or moreprocessors, or by a combination of both. Additionally, these sequence ofactions described herein can be considered to be embodied entirelywithin any form of computer readable storage medium having storedtherein a corresponding set of computer instructions that upon executionwould cause an associated processor to perform the functionalitydescribed herein. Thus, the various aspects of the invention may beembodied in a number of different forms, all of which have beencontemplated to be within the scope of the claimed subject matter. Inaddition, for each of the embodiments described herein, thecorresponding form of any such embodiments may be described herein as,for example, “logic configured to” perform the described action.

The processing unit 21 may also include a main memory, such as a randomaccess memory (RAM) or other dynamic storage device (e.g., dynamic RAM(DRAM), static RAM (SRAM), and synchronous DRAM (SDRAM)), coupled to thebus for storing information and instructions to be executed by theprocessor 22. In addition, the main memory may be used for storingtemporary variables or other intermediate information during theexecution of instructions by the processor 22. The processing unit 21may further include a read only memory (ROM) or other static storagedevice (e.g., programmable ROM (PROM), erasable PROM (EPROM), andelectrically erasable PROM (EEPROM)) coupled to the bus for storingstatic information and instructions for the processor 22.

In some embodiments, the robot may include the ability to detect contactbetween the robot and objects or surfaces in an environment. Thiscontact detection may be implemented by one or more touch sensors, suchas a body touch sensor 35 and/or an arm touch sensor 36. In furtherembodiments, a touch sensor 35, 36, may operate through detection ofdisplacement of a surface on the robot 100, for example the depressionof a bumper, deformation of an electromechanical skin assembly 50 (FIG.5), or other electromechanical detection device that is able to provideelectrical communication upon contact between the device and an objector surface. In further embodiments, a touch sensor 35, 36, may operateby detecting the force applied to a joint on the robot 100 as beinggreater than the force typically required to move the joint. In furtherembodiments, a touch sensor 35, 36, may operate by detecting the forcethat an actuator on the robot is exerting, e.g. by directly measuringtorque or e.g. by measuring electrical current required by the actuatoror e.g. by measuring the error between the target position and theactual position of the actuator. In further embodiments, a touch sensor35, 36, may comprise an optical or ultrasonic sensor, which may be ableto detect contact between a portion of the robot 100 and objects orsurfaces in an environment by detecting very close proximity.

In some embodiments, detecting the force that an actuator on the robot100 is exerting may be accomplished by measuring the current flowingthrough a group of motors 34 or actuators in an arm 200, examples ofwhich is shown in FIGS. 1 and 4, at the ground connection to determinethe force being applied by that set of actuators as a set. FIGS. 1 and 4show an example of an arm 200 attached to the body 11 of a robot 100according to various embodiments described herein. An arm 200 maycomprise an arbitrary number of joints, represented in this illustrationby hinge joints 201 with one or more arm motors 34 controlling themovement of one or more hinge joints 201. In this example, the arm 200is applying force 299 in the direction shown, with resisting force beingapplied in equal magnitude in the opposite direction. An arm 200 maycomprise any number, such as one, two, three, four, five, six, seven, ormore joints 201, arm motors 34, and/or sections 206. It is to beunderstood that the arms 200 described herein are exemplary and that anyother type or style of articulated or non-articulated arm may be used,including arms that move through telescoping sections instead of throughrotating joints.

In some embodiments, perception of surfaces and objects in anenvironment may be accomplished using touch data, provided by one ormore touch sensors 35, 36, and/or using distance data, provided by oneor more distance sensors 37. In further embodiments, the robot 100 maybuild a data structure that represents the positions where it hasdetected obstacles, e.g. surfaces and objects using touch data and/ordistance data. This data structure may be a map of the local environmentand is sometimes referred to herein as a “map”. One representation ofsuch a map is as a 2 or 3 dimensional array of numbers that indicatelikelihood that a particular feature (such as a wall) is in a particularposition (represented by the array indices). The map may take on otherforms as known to those skilled in the art, such as an array or linkedlist of non-zero locations in the map.

In some embodiments, a map of the local environment is created fordifferent poses (i.e. position and orientation) of the robot 100. Forexample, the robot 100 could be in one pose and use touch data receivedby a touch sensor 35, 36, to map the area reachable by a movable arm200, and then turn or move to a different pose and map the new area thatis reachable by the arm 200. In some embodiments, these maps may befused together for example by using dead reckoning (such as how far thetransportation conveyances 12 (FIG. 1), such as wheels, turned whenmoving the robot 100) to figure out the approximate relationship betweenthe maps, with fine adjustment of the maps to each other performed bymatching overlapping features of the two maps using, for example, amotion estimation algorithm as is well-known to those well-versed in theart of image processing.

In some embodiments, maps may be built using a combination of touchsensors 35, 36, and non-touch sensors such as distance sensors 37. Forexample, a distance sensor 37 may comprise a 3D laser distance sensorwhich may be used to provide distance data which may be used to map bothfar 402 (FIG. 9) and near objects 401 (FIG. 1). Touch data from a touchsensor 35, 36, could be used to map near objects 401, 402. The map ofthe near objects 401, 402, could include information that is notavailable to the distance sensor 37, such as the shape of the backsideof an object 401, 402, or information about texture, such as where groutor where bits of raised dirt are. Conversely the distance sensor 37could provide data not available through touch, such as the position offeatures that are beyond the reach of the touch sensor 35, 36, or dataon features that have not yet been touched. In some embodiments, thefeatures from the data provided from the touch and non-touch sensors areplaced on separate maps which are then treated as one fused map througha combination of knowledge of the positional calibration between thetouch 35, 36, and non-touch 37 sensors and/or by aligning the maps basedon features that appear on both maps. In particular, rough fusing canoccur through the knowledge of the positional calibration of the touch35, 36, and non-touch 37 sensors, with fine adjustment of the relativepositions of the maps occurring through matching features that appear onboth maps.

In some embodiments, touch 35, 36, and non-touch 37 sensors may directlyplace their data on the same map. In some embodiments, data that comefrom touch 35, 36, and non-touch 37 sensors may be tagged as touch basedand non-touch based, respectively. For example, data from a touch-basedsensor 35, 36, may be tagged as being touch-based data and data from anon-touch based distance sensor 37 may be tagged as being non-touchbased data.

In further embodiments, the position and pose of the robot 100 may bedetermined by touching the objects 401, 402, around it to obtain touchdata using touch sensors 35, 36. The position and structure of theobjects 401, 402, may be simultaneously determined as the touch data isobtained. This may be done using techniques known to those versed in theart such as Bayesian inference. Bayesian inference can be implementedthrough genetic algorithms (such as a particle filter) orexplicitly-defined predictive modeling (such as Kalman filtering).Distance or non-touch based data and data from touch sensors 35, 36, ortouch based data may be fused into a single data model with Bayesianinferences using the statistical expectation of their accuracy andprecision to combine them. In other embodiments, the touch based dataand non-touch based data may be fused using any other type ofstatistical inference such as frequency inferences, propensityinferences, or state-of-belief inferences.

Use of particle filters and other genetic algorithms to map a two orthree dimensional space for the purpose of navigation may be referred toas SLAM, which stands for Simultaneous Localization and Mapping. SomeSLAM implementations use a rotating laser distance sensor that reads acontinuous stream of distances in various directions. SLAM can also beapplied using data from distance sensors 37 such as 3D depth sensors orfeature classification of RGB video.

In some embodiments, of robotic touch perception described herein, aparticle filter may be used to generate a statistical likelihood modelof the position and the orientation of the robot 100. In the art thecombination of position and orientation is called a “pose”. When using aparticle filter a guess as to the pose of the robot 100, such as thepose of the body 11 or arm 200, is called a “particle”. For each ofthese particles the distance sensor 37 distance data or proximity dataare graphed on a map. Particles for which successive distance data areconsistent with the map are deemed to be more likely to be correct so aprobability counter is increased, while those that show inconsistentdata are deemed less likely to be correct so a probability counter isdecreased. When the probability counter is below a cut-off value theparticle is deemed to be confirmed incorrect and replaced with a newparticle. At the same time, positions that the distance data most oftenidentified as unchanging in the environment, such as walls andstationary objects 401, 402, (which from now on we will refer to asobstacles) are deemed to be more likely to be actual obstacles so aprobability counter associated with having an obstacle in that positionis increased, whereas positions that don't consistently have distancereadings land on them are deemed to be less likely obstacle positions sothe probability counter associated with having an obstacle in thatposition is decreased.

Very quickly a fairly good map of the environment comprising obstaclesemerges as well as a very good guess as to the pose of the robot 100.The resulting map of obstacles along with the most likely pose may beused to navigate the robot 100 through its environment. All of this isexisting art and is well-known to anyone in the field. Mathematicianswould describe this general type of problem as inferring the solution totwo problems simultaneously—in this case the location of the obstaclesand the pose of the robot. Mathematicians would refer to the class ofsolution that the particle filter belongs to as anExpectation-Maximization (EM) algorithm. There are other techniques thatfall in the general category of EM algorithms that may equally be usedto solve this class of problems, such as Kalman Filters. Further, theremay be other Bayesian inference techniques that may be used as well. Thefollowing descriptions are based on particle filters, but those skilledin the art will recognize that any other EM algorithm, Bayesianinference algorithm, or any other technique that can solve two or moreprobabilistic problems simultaneously may equivalently be used.

The problems that must be solved by a robot comprising a movable arm aremore complicated than the simple navigation problem. If such a robot ismobile it may use a distance sensor 37 and standard SLAM to determineits pose and the location of walls and obstacles. However, such a robotmay also build a map of locations of obstacles (e.g. walls and objects)within 3D space. A 3D map described as discrete grid locations may bereferred to as a voxel map in which the elements of such a map arevoxels. Other representations of 3D maps are known to those in the art.For example, a collection of non-discrete coordinates is known as apoint cloud, with each coordinate element being a point. Another 3D mapformat is a collection of polygonal surfaces defined using a unit normalvector and a collection vertices as a surface map. Various forms ofmapping can also add additional information to each element in thecollection such as statistical confidence and the last observed time.For the purpose of illustration the following discussion will be basedon a voxel map, but those skilled in the art will recognize that thediscussion could instead be mapped to any representation of a map, suchas a point cloud, surface map, or other map representation which has notbeen discussed.

A map may include characteristics of the obstacles, such as whether theymove when pressed on. In some embodiments, a map may be created withdistance data captured by a distance sensor 37, such as a depth camera,which is mapped onto a three dimensional map data structure. In furtherembodiments, the pose of the distance sensor 37 may be determinedthrough another means, for example through standard 2D SLAM techniques.One or more EM algorithms may be applied to the distance data from thedistance sensor 37. For example, if the EM algorithms are implemented asparticle filters, for each data point captured by a distance sensor 37,a likelihood counter for the corresponding voxel or other map element inthe two or three dimensional map data structure is incremented.Simultaneously the likelihood counter for voxels or other map elementthat are in the path of the distance sensor 37 but not read as a surfaceby the distance sensor 37 are decreased.

In preferred embodiments, a distance data measurement device distancesensor 37, such as a depth camera, may be mounted on a movable arm 200that is in turn mounted or coupled to a mobile body 11 of a robot 100 asshown in FIG. 1. In further embodiments, standard SLAM techniques may beused to determine the pose of the body 11, and the measured position ofjoints 201 on the arm 200, such as by using rotary encoders that measurethe position of each joint 201, may be used to determine the pose of theelements of the arm 200 relative to the body 11. The pose of the body 11may be mathematically combined with the pose of the arm 200 relative tothe body 11 to determine the pose of the arm 200 relative to the overallenvironment.

In some embodiments the pose of the arm 200 relative to the body 11 maybe determined using a technique known as visual servoing, wherein aunique visual mark known as a fudicial is placed on the arm 200 andtracked by a camera.

In some embodiments, a map of the environment and pose of the arm 200may be determined using an EM algorithm. In further embodiments, aparticle filter may be used to create the map and determine the pose ofthe arm 200. For example, particles may be created for likely positionsof the arm 200. One or more tuples of distance to nearest obstacle anddirection may be measured, for example by using a depth camera distancesensor 37 to determine distance to nearest obstacle in many directionsat once or by using a distance sensor 37 to determine distance to thenearest obstacle in a single direction. Each combination of tuple andparticle may be compared to a 3D map of probabilities of position ofnearest obstacle. A measure of particle likelihood may be increased ifthe combination of tuple and particle indicates a position that islikely to be a position of nearest obstacle, and decreased if thecombination of tuple and particle point to a position that is unlikelyto be a position of nearest obstacle. The increase or decrease may beproportional to the likelihood. The probability of a given voxel orother map element being the position of nearest obstacle is increasedproportionally to the number of combinations of tuple and particles thatindicate it is a voxel or other map element that is the position of thenearest obstacle. Voxels or other map elements that have no tuple andparticle combinations indicating that it is the position of the nearestobstacle may have their probability measure decreased. Particles thatare deemed to be very unlikely may be replaced with particles that havea pose that are deemed to be more likely. These steps may be repeatedover and over again, in a preferential embodiment once per distancesensor 37 reading. The pose of the robotic arm 200 may be deemed to bethat of the particle with the highest likelihood measure. In someembodiments, the particle probability is reported to higher levels ofthe software. In other embodiments, techniques other than particlefilters may be used and be functionally equivalent.

In some embodiments decisions may be made according to the probabilityof correctness of the highest likelihood particle for pose of the robot100 and/or arm 200. This decision could include one or more of thefollowing: decision as to whether to continue a first operation (such aswiping) or to perform a second operation (such as calibration) afterwhich the first operation is returned to; a decision as to whether topause to allow particle likelihoods to settle due to continued sensorreadings allowing refinement of the map; and/or a decision to abandoncurrent operation because position determination is uncertain.

For standard 2D SLAM the selection of initial and replacement particlesmay be accomplished by a suitable methodology. For example, if theposition of the robot 100 is thought to be known, particles are chosenfor robot 100 position and orientation that are the same and almost thesame as the robot 100 had in the most recent scan, as well as particlesfor positions and orientations that are predicted for the current SLAM,e.g. by applying the motion vector expected by the turning of therobot's drive wheels or other transportation conveyances 12. If therobot 100 position is not known, typically particles are created forsome number of random positions and orientations within the possiblespace.

In some embodiments, a method of determining the three dimensional poseof the arm 200 may include creating particles that represent theposition of the arm 200 based on the springiness or resilience of thearm 200, such as by calculating approximations of possible positions ofthe end of the arm 200 if it is bouncing up and down and/or side to sidedue to lack of rigidity, optionally based on a physics-basedmathematical model of the robot arm

In some embodiments, a method of determining the three dimensional poseof the arm 200 may include creating particles based on the end of thearm 200 getting stuck. For example, the arm 200 may be commanded to movebut the portions of the arm may be unable to move, so the robot body 11may be moved e.g. by rotating.

In some embodiments, distance measurements are performed using touchbased data provided by one or more touch sensors 35, 36. One or moretouch sensors 35, 36, may be are located anywhere on the robot 100, suchas anywhere along the arm 200 and/or body 11. A touch sensor 35, 36, maybe any sensor that detects close proximity or mechanical contact betweenparts of the robot 100 and its environment. This is understood to alsoinclude sensors that indirectly detect touch, proximity, and/or contactby measuring the consequence of touch, such as a sound sensor thatdetects the sound of a robotic arm 200 hitting a surface, or anaccelerometer whose data can be examined for acceleration spikes thatare indicative of contact with a surface, or a pressure or strain gaugethat would have a similar spike in its reported data when contacting asurface. Touch sensors 35, 36, may also include but are not limited tocapacitive touch sensors, pressure sensors (such as Force-SensitiveResistors and load cells), and/or short-range binary proximity sensors,such as the commonly-available optical proximity sensors.

In embodiments, a map of the environment and pose of the arm 200 and/orbody 11 may be determined using touch data provided by one or more touchsensors 35, 36. For example, particles may be created for likelypositions of the arm 200. Expected direction of motion for each touchsensor 35, 36, may be calculated based on arm motor 34 velocity for eachof the arm joints 201 and/or on the body motor 33 velocity for the body11 transportation conveyances such as drive wheels. The expecteddirection of motion may be combined with a binary measure of whether ornot the touch sensor 35, 36, is detecting touch with an object 401, 402.This direction and the binary measure may be combined into a tuple,which may be used in exactly the same manner as described earlier fordistance-sensor based tuples.

In some embodiments, tuples based on touch sensor 35, 36, data andtuples based on distance sensor 37 distance data may be processed usingthe same 3D obstacle probability map. In further embodiments, tuples maybe assigned reliability score based on multiple factors. For example,touch data from one or more touch sensor 35, 36, may be deemed to bemore accurate and reliable than distance data provided by one or moredistance sensors 37. In such a case the touch data tuples may incrementprobabilities at a factor that is scaled up from the probabilityincrements from distance data tuples, for example each touch tuple mayaffect probabilities on the map with twice the change the distance datatuples. The probability scaling factor may be different based ondifferent circumstances. Such circumstances may include position ofsensor 35, 36, 37. For example, a touch sensor 35, 36, on the body 11could have a very high probability scaling factor because its positionis well known, whereas a touch sensor 35, 36, on the end of an arm 200could have a lower probability scaling factor because its position ismore uncertain. Such circumstances may also include position of roboticarm 200. For example, if the robotic arm 200 has more accurate detectionof its position when in a folded position than when in an extendedposition the probability scaling factor could be changed accordinglybased on the position of the arm 200.

In preferred embodiments, one or more of the following exemplary sensorconfigurations may contribute to a single 3D obstacle probability map ofthe environment, with desired probability scaling factors: a depthcamera distance sensor 37 mounted on the robotic arm 200; a set of armtouch sensors 36 mounted to the end of the robotic arm 200 that measurepressure applied to an accessory 300, such as a cleaning pad or othersurface interface, mounted to an accessory mount 221; a set of arm touchsensors 36 along the arm 200, including one at one or more joints 201; aset of body touch sensors 35 on the periphery of the body 11; and/or abinary sensor that detects a drop off of the floor, such as would occurat a flight of stairs.

In some embodiments the robot is moving the end of its arm across asurface, for example to wipe a surface. In further embodiments, apressure sensor 35, 36, may be multi-dimensional and may be used tomeasure the pressure during the movement. A multi-dimensional pressuresensor 35, 36, could detect pressure in one or more of the followingdirections which for illustrative purposes are described relative to awiping pad that has pressure applied normal to a surface. For example, amulti-dimensional pressure sensor 35, 36, could be coupled to anaccessory mount 221 to which an accessory 300 such as a cleaning orpolishing pad may be attached. The multi-dimensional pressure sensor 35,36, may then detect pressures such as normal pressure to the top orbottom of the pad, normal pressure to the left side or right side of thepad, pressure to rotate the pad to the left or the right, pressureparallel to the upper or lower edge of the pad, pressure parallel to theleft or right edge of the pad.

In some embodiments, a control loop may be used to control pressure ofthe end of the arm 200 as it moves across a surface. Any type of controlloop may be utilized. A common type of control loop forphysically-actuated systems is known as a PID loop for Proportional,Integral, and Derivative, however there are many other kinds of controlloops that may be used equally well. In further embodiments, the robot100 is configured to move an accessory 300, such as a flat pad, otherflat object, or rotating scrub brush, which may be secured to anaccessory mount 221 on an arm 200 and which may be normal to an objectwith compound curves by using a control loop such as a PID loop using aset of touch sensors 36, such as those detecting pressure normal to thetop/bottom and left/right of the pad. In preferred embodiments, a padmay have three touch sensor inputs that detect pressure normal to thepad. These three sensors may form a triangle. Through simple arithmeticthese three sensors 36 can be used to calculate pressure normal to theupper part of the pad, the lower part of the pad, the left part of thepad, and the right part of the pad.

In a control loop, such as that discussed above, an error (desired valueminus actual value) is calculated which a controller attempts tominimize by changing the state or position of an active element, such asa motor 33, 34. In the control loop described above the error is thedesired pressure minus the actual pressure in each of the directions andthe controller controls the motors to minimize this error. In someembodiments a control loop error may be used as a touch sensor reading.For example, a control loop error that shows expected or high pressurerepresents a reading that indicates the surface is being touched, whilean error that indicates pressure a specified amount below targetpressure (such as zero pressure) represents a reading that indicates thesurface is not being touched. In other embodiments, touch data may comefrom one or more of the following: error readings from control loopsused for wiping a pad or probe, touch data from touches that areexplicitly made for the purpose of determining where obstacles are;incidental and unexpected positive touches, such as the arm 200indicating touch when it was expected not to; lack of touch when touchwas expected, such as the arm 200 not detecting touch when it is thoughtthat it has been moved into position touching a surface such as a table.

In some embodiments the 3D map includes both probabilities of a locationcontaining an obstacle, as well as other visual information, which couldinclude but is not limited to: brightness, saturation, and/or hue (aswould be measured directly by a distance sensor 37 such as camera);transparency (e.g. change in brightness, saturation, and/or hue whenlight is shined behind the object, either directly, or by bouncing lightoff of a surface or object behind the object); and/or variation (e.g.standard deviation of measured brightness, saturation and/or hue fromreading to reading).

In some embodiments a 3D map of touch data characteristics may becreated. This map could include whether the surface moved when touched,optionally along with how much force was required, optionally along withwhether the surface returned to the original spot when pressure wasreleased. Such detection could be performed using the following steps:move a robotic arm 200 with a pressure sensor 36 attached towardssurface until pressure is first detected; attempt to move the roboticarm 200 further in the same direction until either a specified maximumallowable pressure is reach or a specified maximum allowabledisplacement is reached; return the robotic arm 200 to the position offirst pressure detection, measuring the position of the robotic arm 200where pressure falls to zero. Those skilled in the art will recognizethat there are other sequences of motion of the robotic arm 200 thatwill yield similar results and are functionally identical to thissequence.

In some embodiments, a 3D map of touch data characteristics such assurface texture detection. Such detection could be performed using thefollowing steps: move a probe or pad accessory 300 across a surface witha constant force normal to the surface; measure the lateral forcerequired to move the pad or probe; optionally, repeat the process with adifferent normal force. The lateral force may be proportional to theroughness of the surface, with smooth surfaces requiring little lateralforce to move the pad or probe, whereas rough surfaces may require morelateral force. The shape of the probe may be chosen to exclude roughnesscaused by small features. If the probe is a sphere, small spheres maydetect roughness including both small and large features. Largerspherical probes may exclude roughness due to features that aresubstantially smaller than the radius of the sphere. Various probeshapes may be chosen. Symmetrical shapes such as a sphere will detecthorizontal and vertical roughness equally, whereas a non-symmetricalshape such as a rectangle with a rounded edge may preferentially detectroughness in a single direction. Such a probe could be useful fordetecting roughness due to e.g. grout lines in bathroom tile.

In some embodiments the 3D map of touch data characteristics is usedtogether with either or both of the 3D map of probabilities of obstacleand/or the 3D map of visual characteristics. In some embodiments a 3Dmap of places where the arm 200 got stuck or was unable to move iscreated. This may be a 3D voxel map which may be combined with one ormore of the previously described 3D maps.

In some embodiments, changes in surface material or different types ofmaterial may be detected by touch sensors 35, 36, and special cleaningalgorithms that also utilize touch may be applied. For clarity theexample presented below talks of grout, but other changes in surfacematerial apply equally well, such as caulk, mortar in bricks, etc. Forthe purpose of this explanation, an area which has surface dirt thatmakes an otherwise smooth surface feel rough, such as dirty tiles, maybe considered a change in surface material (dirty tile versus clean) forthe purposes of this description. In preferred embodiments, thealgorithm may comprise a robot performing moving a probe across thesurface of interest. The probe may comprise an object that has a shapethat would cause it to “catch” on one of the materials, for examplegrout, by providing a change in resistance when it moves from thesurface of e.g. a tile to grout. An example of such a shape would be arod with a hemispherical end, which would descend into a grout line ifmoved across the surface of tiles that are separated by grout. Anotherexample of such a shape is a bristled brush, which would provide moreresistance to lateral motion on a rough surface than exists on a smoothsurface. A constant pressure normal to the surface may be maintained.When a surface change (e.g. a grout line) is crossed, the force requiredto move across the surface will change, and will then return to itsoriginal value when back on the first material (e.g. tile).

In some embodiments, the probe may be moved across the surface in two ormore directions of travel. The position where force begins to change islikely to be after the beginning of the change in material (e.g. grout),and the position where the force begins to return to the previous valueis likely to be after the end of the change in material (e.g. grout).The force curves for the different directions versus position may becombined using a physics-based model that takes into account probeshape, shape of the area where material changes (e.g. that tile is proudof grout), and/or material properties (e.g. that grout is rougher andtherefore requires more force to move a probe across it). By running twoor more directions over the change in material (e.g. grout) the actualposition of the grout may be determined more accurately.

In some embodiments, a visual system may be used help identify materialchange (e.g. grout) position. Material change position detected bychange in force may be compared to edge detection data from the visualsystem, and if an edge is present the visual system edge may be used tocorrect the material change (e.g. grout) position. In some embodiments,lines (e.g. straight lines) may be fitted to the visual data and/orprobe-based data using standard curve-fitting techniques known to thoseversed in the art of mathematics. In further embodiments, a map of thematerial changes (e.g. grout lines) may be made.

In some embodiments, a secondary material that sits below the surface ofthe primary material (for example grout sitting below the surface oftile) may be cleaned. In preferred embodiments, this cleaning may beperformed by an accessory 300 which may function as a probe, such as acleaning pad, cloth, rotating brush, or other cleaning device that maybe attached to a robot 100 such as to the distal end 212 of an arm 200.In some embodiments the edge of the cleaning pad, cloth, rotating brush,or other cleaning device may be turned to get into the well ordepression that the secondary material forms. In further embodiments, aback and forth scrubbing motion may be performed. In still furtherembodiments, a pad or cloth has vibration applied to it to improvecleaning. In further embodiments, the secondary material (e.g. grout)may be cleaned with a rotating brush.

FIG. 5 depicts a perspective view of an example of a touch sensorcomprising electromechanical skin assembly 50 according to variousembodiments described herein. In some embodiments, collision between therobot 100 and other things (including but not limited to objects, walls,and elements of the robot 100 itself) may be detected through a flexiblepressure sensitive electromechanical skin assembly 50. Such a skinassembly 50 may be constructed with an electrically conductive skinmaterial 51, such as an electrically conductive silicone or rubber,stretched so that it is just proud of a conductive surface 52 as anexample. In some embodiments, the conductive surface 52 of the skinassembly 50 may be formed by stretched strings 53 of conductivematerial, for example conductive rubber in the shape of rubber bands.The geometry may be designed such that deformation of the skin material51 results in touching the conductive surface 52, completing anelectrical circuit. Detecting the position of deformation could beaccomplished by segmenting either conductive skin (e.g. by havingalternating strips of conductive and non-conductive material) or bysegmenting the conductive surface. If the skin 51 is segmented in onedirection (for example vertically) and the conductive surface 52 may besegmented in another direction (for example horizontally) then fullposition information of where the deformation will be available. Amountof force or area related to the deformation may be detected by havingconductive skin material 5 lthat has resistance at the contact point, sothat the resistance of the connection would indicate how much skinmaterial 51 is touching the conductive surface 52.

In some embodiments, an electromechanical skin assembly 50 configured toprovide touch data may also be constructed by sending an AC signalthrough the skin 51 that is a frequency that would resonantly couplewith maximum efficiency with the underlying surface at a particulardistance. In this case full contact between the two surfaces 51, 52, isnot required, as the amount of coupling is determined by the distance ofthe two surfaces 51, 52, from each other, which in turn is affected bythe pressure that the skin assembly 50 has applied to it.

Turning now to FIGS. 1, 6-8, in preferred embodiments, a robot 100 maybe configured to move an accessory 300, such as a flat pad, other flatobject, or rotating scrub brush, which may be secured to an effectorsuite 220 as shown in FIG. 1. An effector suite 220 may be coupled tothe arm 200, such as to the distal end 212, and may be configured tosecure an accessory 300. In some embodiments and as shown in FIGS. 6-8,an effector suite 220 may comprise one or more arm touch sensors 36,distance sensors 37, and/or accessory mounts 221. Generally, anaccessory mount 221 may be configured to couple an accessory 300 to theeffector suite 220 with the accessory 300 typically configured tocontact or interact with objects 401, 402, and surfaces in theenvironment. For example, an accessory mount 221 may be configured tosecure a cleaning pad accessory 300 as shown in FIG. 1 which may bemoved to contact or wipe the surface of an object 401.

Accessory mounts 221 may be configured in a plurality of shapes andsizes to allow a plurality of types of accessories 300, such as pads,brushes, probes, air jets, water jets, buffing pads, polishing brushes,grout cleaning brushes or pads, grasping or gripping devices, or anyother type of accessory which may be moved into contact with an object401, 402. In preferred embodiments, an accessory 300 may be moved intocontact with an object 401, 402, by the robot 100 and/or arm 200 toallow the robot to complete a goal directed action such as cleaning,polishing, drying, washing, buffing, sanding, dusting, painting, or anyother type of action or activity. For example, a cleaning brushaccessory 300 may be secured to the accessory mount 221 of an effectorsuite 220 which is coupled to the distal end 212 of an arm 200 of arobot 100. The robot 100 may move into proximity of an object 401, 402,and then move the arm 200 to rub the cleaning brush across the object,thereby completing the goal directed action of cleaning a portion of theobject 401, 402.

In preferred embodiments, an accessory mount 221 may be coupled to anarm touch sensor 36 and configured to receive touch data when theaccessory mount 221 and/or an accessory 300 secured to the accessorymount 221 is moved into contact with an object 401, 402, or surface. Forexample, an accessory mount 221 (FIG. 6) may be coupled to a touchsensor cover 222 (FIG. 7) which is configured to shield or cover one ormore, such as three, arm touch sensors 36. These three sensors 36 may bepositioned in a generally triangular shape as shown in FIG. 8 andthrough simple arithmetic these three sensors 36 can be used tocalculate pressure normal to the upper part, lower part, the left part,and the right part of the mount 221 and/or an accessory 300 secured tothe accessory mount 221 is moved into contact with an object 401, 402,or surface. In other embodiments, an effector suite 220 may comprisethree touch sensors 36 positioned in a generally triangular shape whichmay be formed by three force sensitive resistors and a touch sensorcover 222 which may have three protrusions which are each positioned tocontact a force sensitive resistor. An accessory mount 221 may becoupled to the touch sensor cover 222 thereby allowing the touch sensor36 to receive touch data when an accessory 300 secured to the accessorymount 221 contacts an object 401, 402, thereby causing one or moreprotrusions on the cover 222 to interact with one or more forcesensitive resistors.

FIG. 9 illustrates two example scans of an object using discrete twodimensional distance sensor 37 whose scans may be used for mapping of anenvironment according to various embodiments described herein. In someembodiments, the scans may be made by one or more distance sensor 37which may be positioned on an effector suite 220 (FIGS. 1, 6-8) orpositioned anywhere on a robot 100. Example cases of a scan from theside or from a different angle are shown in FIG. 9. FIG. 9 shows twoexample scans of a generally cylindrical object 402 which may be madeusing discrete two dimensional distance sensors 37A, 37B or made bymoving a single distance sensor 37 into two different positions A,B, andwhose scans follow the path 603 and 604 respectively, resulting indiscrete scan readings or distance data approximating an ellipse 605 forsensor 37A in position A and a circle 606 for sensor 37B in position Baccording to various embodiments described herein. In this embodimentthe position points along an object 402 could be compared to the valuesthat would be generated by the combination of primitive shapes in eachlibrary entry utilizing the range of possible parameter values specifiedin the library entry, searching a database in a data store 308 (FIG. 11)to see if there is a set of parametric values that yield an error (suchas mean squared error) below a threshold value. Parametric values mayinclude zero or more of size, diameter, size or shape of a fillet, etc.The threshold value may either be an absolute number, or may be scaledaccording to relevant factors, such as the size of the object, or howmuch variation that type of object 402 might have (for example aparticular size lego brick is manufactured to close tolerances, whereasa bar of soap may be a large variety of different sizes). Choosingparameter values could use techniques that are well-known in the art forfinding global minima, such as random seed starting points followed by adirected search for minima starting from that seed.

In some embodiments, multiple parallel scans of the same space may bemade, for example horizontally at heights of 1 inch, 3 inches, and 5inches from a reference point, such as the largest flat surface toprovide distance data for use with a library of objects 502 (FIG. 12).Optionally, the library 502 may contain the expected values of all threescans in alignment. In some embodiments, the expected values of one ormore perpendicular scans may be included in the library 502. In thisembodiment scans may be made in one direction, yielding a set of objectcandidate matches by comparing the scan data against the library data.The matches may be ruled in or out according to how well they match whena perpendicular scan is performed.

FIG. 12 shows a block diagram of an example of an object library 502database of data associated with objects 401, 402, according to variousembodiments described herein. In some embodiments, a library of objectsand/or surface shapes 502 exists. The library 502 may include a datarecord 501 for one or more, and preferably for each object 401, 402,that has been or may be encountered in an environment. The library 502may be in the form of a database which may be stored in a data store 308accessible to the processing unit 21 of the robot 100. Optionally, thedata store 308 may be locally accessible to the processing unit 21and/or remotely accessible such as a cloud based data store 308accessible through the radio 23 or other network interface. Each datarecord 501 may comprise one or more data fields or categories, such as aname category 511, a shape data category 512, a parameters that maychange category 513, an attributes category 514, a reaction to stimulicategory 515, a center of gravity category 516, a grasping category 517,a cleaning category 518, an attachment category 519, and/or any othercategory or data field which may receive or store data associated withan object 401, 402.

Scans of objects 401 (FIG. 1), 402 (FIG. 9), (through touch, 3Drange-finding, or other 3D detection techniques) may be matched toobject data records 501 within the library 502. The library 502, shownin exemplary form in FIG. 10, may contain additional information aboutthe objects 401,402, for example whether they are fixed (like a faucethandle) or movable (like a shampoo bottle) which may be stored in aattributes category 514, what material they are made of which may bestored in category 514, the motions necessary to traverse the surface,for example for cleaning, how to grasp for moving the object 401,402,etc which may be stored in categories 515-518. The library 502 may alsoinclude information about the object 401,402, that represents theobject's reaction to stimulus which may be stored in category 515, suchas whether it moved when a force was applied to the top, orcharacteristics that required both interaction and calculation todetermine, such as the center of gravity of the object 401,402, whichmay be stored in category 516 or how full a shampoo bottle is.

There are a number of ways that objects 401,402, may be matched toobjects in the library 502. In some embodiments, objects 401,402, may bematched to object data records 501 in the library 502 as an exact match.The library 502 would include a set of data in a data record 501 thatrepresents each object's shape, and position points along the unknownobject 401,402, would be compared with the library of shapes with someerror allowed. For example the position points along the unknown object401,402, could be compared to the shape categories 512 in the library502, and a least mean square error generated between the position pointsalong the unknown object 401,402, and the shape in the library 502, witha match declared to the library shape with the lowest least mean squareerror as long as the error is below a threshold.

In some embodiments, objects 401,402, may be matched to data records 501of objects in the library 502 as parametrically as a combination ofprimitive shapes. The primitive shapes could include but are not limitedto cylinders, cones, rectilinear volumes with or without roundedcorners. For example the library 502 could determine that a set ofposition points along an object 401,402, that is consistent with beingcylindrical is a bottle. Or it could be more specific—any object401,402, that has a lower portion that is consistent with beingcylindrical with a diameter of 2 to 5 inches, and an upper portion thatis consistent with being cylindrical with a smaller diameter is ashampoo bottle. When we speak of “consistent with being cylindrical” werefer to distance sensor data or touch sensor data from above that showsan object 401,402, that is circular when viewed from above, and/or ascan or touch from the side that shows a half cylinder (the other halfbeing obscured by the front of the object) and/or a scan or touch from adifferent angle that shows an ellipse.

In some embodiments, objects 401, 402, may be matched to data records501 of objects in the library 502 as functionally. A set of rules wouldbe considered that form a decision tree for the type of object 401, 402.For example objects 401, 402, that touch a horizontal surface areassumed to be standalone objects like shampoo bottles, and objects 401,402, that do not touch a horizontal surface are assumed to be fixedobjects, like a faucet handle.

In some embodiments, a set of parameters may be stored with the datarecords 501 of objects in the library 502, representing the range ofdifferent versions of the object 401, 402. For example, the positionpoints that a rectangular object 401, 402, would generate could be inthe library 502, along with a parameter that describes what theallowable range of rotation that the object might have (for example, acylindrical object would have a single rotation tested, since rotationdoesn't matter, whereas a square object could be tested for rotationevery 5 degrees across 90 degrees of rotation, optionally with rotationsof greater than 90 degrees ignored since they would be identical torotations of less than 90 degrees. In a further example, the positionpoints that a rectangular object 401, 402, would generate could be inthe library 502, along with a parameter that describes what theallowable range of scaling would be. For example an object 401, 402,that is only available in a single size (such as a shot glass) may havea parameter that would indicate that only comparisons to the referencepoints at 1× scaling should be tested, whereas an object 401, 402, thatis available in a number of sizes, such as a shampoo bottle or winebottle might have a parameter indicating that is could be tested forsizes from for example 0.5× to 3×.

In some embodiments, objects 401, 402, may be tested for being fixed ormoveable by pushing on them a very short distance (such as 0.25 inches).If they provide resistance they are fixed, if they move they are movableobjects. The objects' expected response to, for example, being pushed oncould be included in the library 502 and used as one of the parameters514 for object matching.

In some embodiments, the orientation of a distance data or touch datamay be corrected by assuming that straight surfaces that are close tohorizontal are exactly horizontal and by assuming that straight surfacesthat are close to vertical are exactly vertical. Since objects withhorizontal and/or vertical features are very common, this may reduce thecomputing complexity necessary to match objects 401, 402, tocorresponding objects in the library 502 in many cases.

In some embodiments, motions to interact with an object 401, 402, may beincluded in the library 502. For example for a cleaning application if asurface must be sprayed and wiped, spraying and wiping could be apredetermined set of actions for each library data record 501 orparametric shape. In the same cleaning application, it is possible thatsome objects 401, 402, may be meant to be skipped (like a bar of soap)or moved out of the way (like a shampoo bottle). Instructions for theseother interactions could be included in the library 502, including ifappropriate a set of parameters for the action or the exact poses neededto perform the action.

In some embodiments, objects' 401, 402, shapes may be measured, e.g.using touch sensor data or touch perception and/or a camera and/or adistance sensor 37, either one by one or as a group. In someembodiments, object 401, 402, shapes may be measured before and/or afterbeing manipulated, such as being grasped and/or lifted. In someembodiments, objects' 401, 402, shapes may be matched to shape data fordata records 501 of objects in the library 502.

In some embodiments, object 401, 402, shapes may be matched to either adata record 501 of an object in the library 502 or a category 511-519 ofobjects in the library 502. For example, the library 502 could containan entry that has parameters that match a particular puzzle piece, butthe library 502 may also contain an entry that covers a category, suchas puzzle pieces or hard objects below certain dimensions.

In some embodiments, deformable objects 401, 402, such as those made ofcloth may be included in the library 502. In one embodiment they areidentified by a combination of their shape and by the fact that they aredeformable (e.g. they change shape when force is applied). In someembodiments, one object category is that of deformable objects 401, 402.

In some embodiments, the object 401, 402, and object category in alibrary 502 may include a field that indicates what category the objector object category falls into. The categories may form a data structure,such that an object category (such as puzzle piece) may also belong toanother category (such as hard object 401, 402).

In some embodiments, the robot 100 includes a grasper. The grasper willbe capable of interfacing with objects 401, 402, for example in order tolift them. The grasper may use zero or more of the following techniques:fingers which hold an object 401, 402, by holding a feature of theobject 401, 402, between them; suction; electro-adhesion; other means ofadhesion; a needle, or other piercing instrument; an eating utensil,such as a fork or spoon; and/or a special shape that mechanicallyinterfaces with an object 401, 402, e.g. a collar that is the correctshape to accept a mop handle.

In some embodiments, objects 401, 402, may be placed into physical binsby an autonomous machine, or into places that are designated forparticular objects or categories of objects.

In some embodiments, a representation of the object 401, 402, (such as aphotograph or 3D scan) may be added to the database when an object 401,402, is encountered, along with information such as where the object401, 402, was found, and where the object 401, 402, was placed.

In some embodiments, data about the object 401, 402, (such as itsweight) may be added to the database when an object 401, 402, isencountered, along with information such as where the object 401, 402,was found, and where the object 401, 402, was placed. In furtherembodiments data such as weight is used to report to the userinformation about how much of the contents is remaining, such as warningthe owner that they are almost out of milk, and/or taking an action suchas ordering more milk. In further embodiments the robot 100 uses an RFIDreader to query nearby objects and catalog where RFID-enabled objectsare.

In some embodiments, the robot 100 lifts each object 401, 402,optionally one at a time, within an area that is designated as needingde-cluttering. It may take object 401, 402, shape measurements such aswith a touch sensor 35, 36, and/or distance sensor 37, which then may becompared to information in the object database, by which means theidentity or object category for the object may be determined. It thenmay place the object 401, 402, in a bin or in another type of locationthat is designated for that type of object 401, 402.

In some embodiments, some or all objects 401, 402, may be put back wherethey came from. The database may keep track of where an object 401, 402,was found. In some embodiments, the database may be queried to findinformation about objects 401, 402, such as what they look like or wherethey were found, or where they reside now to provide the ability toremotely view an object 401, 402, to see it's state (for example toexamine from a remote location a recent photograph of a shampoo bottlein the shower to see if more shampoo needs to be bought). In furtherembodiments, the database may be queried to find information aboutobjects 401, 402, such as what they look like or where they were found,or where they reside now to provide the ability to find a missing object401, 402, (for example to find where the missing car keys are). In stillfurther embodiments, the database may be queried to find informationabout objects 401, 402, such as what they look like or where they werefound, or where they reside now to provide the ability to remotely viewan object to see the information that is on it (for example to examine apiece of paper that has a password on it)

In some embodiments, the robot 100 can move a group of objects 401, 402,from one place to another, performing an action during the transition.To do this, objects 401, 402, may be scanned to provide touch dataand/or distance data and matched to an object data record 501 in theobject library 502 as previously disclosed. In one embodiment, one ofthe either the source or destination (or both) location of the objects401, 402, is a structured environment that has spaces that may holdparticular types of objects 401, 402, and the robot 100 may match thelibrary data records 501 of objects to the spaces that are marked asaccommodating said objects 401, 402. For example, a robot 100 may fill adishwasher by moving objects 401, 402, (e.g. dirty dishes) from a sink,after which the object 401, 402, is rinsed, after which the object 401,402, is placed into a suitable location in the destination (e.g. thedish is placed into a slot within the dishwasher rack).

In preferred embodiments, a robot 100 may comprise: a grasper; one ormore sensors 35, 36, 37; an object library 502, wherein the grasperlifts one object 401, 402, at a time and a set of object shapemeasurements are taken by a touch sensor 35, 36, and/or distance sensor37 which are compared to each entry in the object library 502, a bestmatch for the object 401, 402, is determined, and the object 401, 402,is placed into a place that the data record 501 says is the proper placefor an object 401, 402, of that type.

In further preferred embodiments, a robot 100 may comprise one or moresensors 35, 36, 37, including one or more touch sensors 35, 36, and arepresentation of the environment or objects 401, 402, within it such asin a memory or data base 308 accessible to the robot 100, wherein datacollected from a touch sensor 35, 36, is added to the representation ofthe environment or objects within it.

In further preferred embodiments, a robot 100 may comprise one or moresensors 35, 36, 37, including one or more touch sensors 35, 36, whereininformation provided by a touch-based sensor 35, 36, is used to map aportion of the environment and/or objects 401, 402, within it.

FIG. 11 illustrates a block diagram illustrating some modules which mayfunction as software rules engines or programs 28 (FIG. 3) and which maybe performed in the memory 25 (FIG. 3) of a processing unit 21 (FIG. 3)according to various embodiments described herein. In preferredembodiments, the programs 28 may comprise a mapping module 600 which mayinclude a mapping engine 601, a touch engine 602, a distance engine 603,and/or a fusion engine 604 according to various embodiments describedherein. In some embodiments, one or more functions of one or moreengines 601-604 may be performed by one or more other engines 601-604.

A mapping module 600 may be configured to facilitate the transfer ofdata between one or more engines 601-604 and/or to facilitate thetransfer of data between one or more engines 601-604 and a data store308 which may include an object library 502 (FIG. 10). Additionally, themapping module 600 may output data which may be used to move the robot100, arm 200, and/or effector suite 220. In some embodiments, a datastore 308 may be remote from the robot 100. The robot 100 may access thedata store 308 through a wired or wireless network connection.Additionally, two or more robots 100 may contribute to and access datain a data store 308 such as an object library 502 (FIG. 10). In furtherembodiments, a memory 25 local to a robot may function as a data store308.

A touch engine 602 may be configured to receive touch data from one ormore body touch sensors 35 and/or arm touch sensors 36. This touch datamay be received when portions of a robot 100 comprising a touch sensor35, 36, contact an object intentionally or as directed by the processingunit 21, unintentionally without direction from the processing unit 21,and/or from moving potions of the robot 100, such as an accessory mount221 of an effector suite 220 across a surface of an object or obstacle.

Similarly, a distance engine 603 may be configured to receive distancedata from one or more distance sensors 37. This distance data mayinclude scans 605, 606, (FIG. 9) of objects and/or distance data betweena distance sensor 37 or portion of the robot 100 and a surface of anobject or obstacle in the environment.

A mapping engine 601 may be configured to build a data structure thatrepresents the positions where it has detected obstacles, e.g. surfacesand objects using touch data and/or distance data. This data structuremay be a map of the local environment and is sometimes referred toherein as a “map”. One representation of such a map is as a 2 or 3dimensional array of numbers that indicate likelihood that a particularfeature (such as a wall) is in a particular position (represented by thearray indices). The map may take on other forms as known to thoseskilled in the art, such as an array or linked list of non-zerolocations in the map. Touch data from a touch sensor 35, 36, provided bythe touch engine 602 could be used to map near objects 401. Converselydistance data from a distance sensor 37 provided by the distance enginecould provide data not available through touch, such as the position offeatures that are beyond the reach of the touch sensor 35, 36, or dataon features that have not yet been touched.

In some embodiments, once the mapping engine 601 has created a touchdata map and a distance map, the fusion engine 604 may then fuse the twomaps into a single map through a combination of knowledge of thepositional calibration between the touch 35, 36, and non-touch 37sensors and/or by aligning the maps based on features that appear onboth maps. In particular, rough fusing can occur through the knowledgeof the positional calibration of the touch 35, 36, and non-touch 37sensors, with fine adjustment of the relative positions of the mapsoccurring through matching features that appear on both maps. Inalternative embodiments, a fusion engine 604 may receive touch data anddistance senor data and may fuse or combine the data. The fused data maythen be supplied to the mapping engine 601 which may then make a singlemap comprising both touch data and distance senor data.

In preferred embodiments, the mapping module 600 may receive a map ormap data and use it to determine the position of the robot 100 and/orarm within the environment or in relation to an object. A mapping module600 may use a particle filter or any other method of determining theposition of the robot 100 and/or arm within the environment or inrelation to an object. Additionally, the mapping module may beconfigured to move the robot 100 and/or arm 200 based on a received mapor map data and the determined position of the robot 100 and/or arm 200.

FIG. 12 shows a block diagram of an example of a method for combiningsensor data to perform a goal directed action (“the method”) 700according to various embodiments described herein. In some embodiments,the method may start 701 with receiving distance sensor data from adistance sensor 37 in step 702. In some embodiments, step 702 may beperformed by a distance engine 603 which may provide the distance datacomprising distance measurements to a mapping engine 601 and/or a fusionengine 604.

Next, in step 703, touch data may be received by a touch sensor 35, 36.In some embodiments, step 703 may be performed by a touch engine 602which may provide the touch data received by contacting an object orsurface to a mapping engine 601 and/or a fusion engine 604.

In step 704, the distance data and touch data may be combined by theprocessing unit of the mobile robot apparatus 100. In some embodiments,once the mapping engine 601 has created a touch data map and a distancemap, the fusion engine 604 may then fuse the two maps into fused orcombined data comprising a single map through a combination of knowledgeof the positional calibration between the touch 35, 36, and non-touch 37sensors and/or by aligning the maps based on features that appear onboth maps. In alternative embodiments, a fusion engine 604 may receivetouch data and distance senor data and may fuse or combine the data. Thefused or combined data may then be supplied to the mapping engine 601which may then make a single map comprising both touch data and distancesenor data.

Finally, in step 705, the combined distance data and touch data may beused to manipulate the robot apparatus 100 to complete the goal directedaction. In preferred embodiments, the mapping module 600 may receive amap or map data and use it to determine the position of the robot 100and/or arm within the environment or in relation to an object. A mappingmodule 600 may use a particle filter or any other method of determiningthe position of the robot 100 and/or arm within the environment or inrelation to an object. Additionally, the mapping module may beconfigured to move the robot 100 and/or arm 200 based on a received mapor map data and the determined position of the robot 100 and/or arm 200.By moving the robot 100 and/or arm 200, the robot apparatus 100 may bemanipulated to complete a goal directed action such as cleaning,polishing, drying, washing, buffing, sanding, dusting, painting, or anyother type of action or activity. For example, a cleaning pad accessory300 may be secured to the accessory mount 221 of an effector suite 220which is coupled to the distal end 212 of an arm 200 of a robot 200. Therobot 100 may move into proximity of an object 401 and then move the arm200 to rub the cleaning pad across the object, thereby completing thegoal directed action of cleaning a portion of the object 401. Once step705 has been completed, the method 700 may finish 706.

While some materials have been provided, in other embodiments, theelements that comprise the robot 100 such as the body 11, movable arm200, and/or any other element discussed herein may be made from durablematerials such as aluminum, steel, other metals and metal alloys, wood,hard rubbers, hard plastics, fiber reinforced plastics, carbon fiber,fiber glass, resins, polymers or any other suitable materials includingcombinations of materials. Additionally, one or more elements may bemade from or comprise durable and slightly flexible materials such assoft plastics, silicone, soft rubbers, or any other suitable materialsincluding combinations of materials. In some embodiments, one or more ofthe elements that comprise the tool 100 may be coupled or connectedtogether with heat bonding, chemical bonding, adhesives, clasp typefasteners, clip type fasteners, rivet type fasteners, threaded typefasteners, other types of fasteners, or any other suitable joiningmethod. In other embodiments, one or more of the elements that comprisethe robot 100 may be coupled or removably connected by being press fitor snap fit together, by one or more fasteners such as hook and looptype or Velcro® fasteners, magnetic type fasteners, threaded typefasteners, sealable tongue and groove fasteners, snap fasteners, cliptype fasteners, clasp type fasteners, ratchet type fasteners, apush-to-lock type connection method, a turn-to-lock type connectionmethod, slide-to-lock type connection method or any other suitabletemporary connection method as one reasonably skilled in the art couldenvision to serve the same function. In further embodiments, one or moreof the elements that comprise the robot 100 may be coupled by being oneof connected to and integrally formed with another element of the robot100.

Although the present invention has been illustrated and described hereinwith reference to preferred embodiments and specific examples thereof,it will be readily apparent to those of ordinary skill in the art thatother embodiments and examples may perform similar functions and/orachieve like results. All such equivalent embodiments and examples arewithin the spirit and scope of the present invention, are contemplatedthereby, and are intended to be covered by the following claims.

What is claimed is:
 1. A method for completing a goal directed actionwith a robot apparatus using robotic touch perception, the methodcomprising: receiving, at a processor, distance data from a distancesensor, the distance data including measurements of distances betweenthe robot apparatus and objects in an environment; receiving, at theprocessor, touch data from a touch sensor, the touch data includingmeasurements of results of the robot apparatus contacting the objects;creating, by the processor, maps of the environment by combining thedistance data and touch data, the maps representing positions of theobjects in the environment, wherein one map of the maps is associatedwith a pose of the robot apparatus, the pose determined by: determininga set of candidate poses using the distance data and the touch dataassociated with the one map, each candidate pose associated with alikelihood that, under the candidate pose, the distance data related toan obstacle in the environment is consistent with the one map, andselecting the candidate pose associated with the highest likelihood asthe pose of the robot apparatus; and controlling, by the processor, therobot apparatus to complete the goal directed action using the maps,comprising: determining, based on the pose, a position of the robotapparatus in relation with an object of the objects, and controlling therobot apparatus to perform a set of motions towards the object.
 2. Themethod of claim 1, wherein the positions of the objects are associatedwith probabilities of the objects present at the positions.
 3. Themethod of claim 1, wherein the goal directed action is selected from oneof; contacting an object and avoiding contact with an object.
 4. Themethod of claim 1, wherein the distance data and touch data are combinedusing Bayesian inference according to statistical expectation ofaccuracy and precision associated with the distance data and touch data.5. The method of claim 1, wherein the robot apparatus comprises a probecoupled to the touch sensor, the method further comprising moving theprobe across a surface to generate the touch data.
 6. The method ofclaim 1, wherein the robot apparatus comprises a probe coupled to thetouch sensor, the probe configured to receive touch data, the methodfurther comprising detecting the object using the distance data andmoving the probe into contact with the object.
 7. The method of claim 1,further comprising: receiving touch data associated with anunrepresented object that is unrepresented in the maps; updating themaps to include the unrepresented object.
 8. The method of claim 1,further comprising: recognizing the objects by matching the distancedata to an object library, the object library storing information aboutobjects.
 9. The method of claim 1, wherein the robot apparatus comprisesa body connected to an arm and wherein controlling the robot apparatusto perform the set of motions towards the object comprises: moving therobot apparatus such that the arm contacts the object and controllingthe arm to perform the set of motions.
 10. The method of claim 1,wherein one map of the maps is three-dimensional and includes surfacecharacteristics of the object, wherein controlling the robot apparatusfurther comprises: determining the set of motions according to thesurface characteristics of the object.
 11. The method of claim 10,wherein the surface characteristics of the object includes at least oneof: a deformation amount of a surface of the object, a force associatedwith the deformation amount, an elasticity of the surface of the object,or a surface texture.
 12. The method of claim 1, further comprisingreceiving updated touch data including surface information of theobject; wherein controlling the robot apparatus further comprises:determining an amount of force applied to the object based on thesurface information, wherein the set of motions are performed to applythe amount of force on the object.
 13. A robot apparatus for completinggoal directed actions, the apparatus comprising: a power source; a body;an arm operably connected to the body with said arm having a first touchsensor configured to generate touch data, the touch data includingmeasurements of results of the robot apparatus contacting objects in anenvironment; a distance sensor configured to generate distance data, thedistance data including measurements of distances between the robotapparatus and the objects; a processor configured to execute one or moreinstructions, the instructions, when executed by the processor, causethe processor to: create maps of the environment by combining the touchdata and the distance data, the maps representing positions of theobjects in the environment, wherein one map of the maps is associatedwith a pose of the robot apparatus, the pose determined by: determininga set of candidate poses using the distance data and the touch dataassociated with the one map, each candidate pose associated with alikelihood that, under the candidate pose, the distance data related toan obstacle in the environment is consistent with the one map, andselecting the candidate pose associated with the highest likelihood asthe pose of the robot apparatus, and determine, based on the pose, aposition of the robot apparatus in relation with an object of theobjects, and control the arm to perform a set of motions towards theobject.
 14. The apparatus of claim 13, further comprising a motorconfigured to operate a transportation conveyance, and thetransportation conveyance configured to facilitate the movement of therobot apparatus across surfaces.
 15. The apparatus of claim 13, furthercomprising a second touch sensor positioned on the body.
 16. Theapparatus of claim 13, wherein the first touch sensor compriseselectromechanical skin.
 17. The apparatus of claim 13, furthercomprising non-transitory memory configured to store the maps.
 18. Theapparatus of claim 13, wherein creating the maps by combining the touchdata and the distance data comprises: building a first map using thedistance data, building a second map using the touch data, determiningmatching features present on the first map and the second map, andcreating a third map by combining the first and second maps through thematching features.
 19. The apparatus of claim 13, wherein the distancedata and touch data are combined using Bayesian inference according tostatistical expectation of accuracy and precision associated with thedistance data and touch data.
 20. The apparatus of claim 13, wherein thearm further comprises a probe coupled to the touch sensor, wherein theprocessor is further configured to cause the probe to move across asurface to generate the touch data.
 21. The apparatus of claim 13,wherein the instructions further cause the processor to create an objectlibrary, the object library storing information about objects.
 22. Therobot apparatus of claim 13, wherein the one map of the maps isthree-dimensional and includes surface characteristics of the object,and the surface characteristics of the object includes at least one of:a deformation amount of a surface of the object, a force associated withthe deformation amount, an elasticity of the surface of the object, or asurface texture.